feat(sinks): log response body when HTTP response failed #24434

keuin · 2026-01-01T09:23:18Z

Summary

This patch adds HTTP response body peek logic, reading for at most 1024 UTF-8 characters from HTTP response. It uses this utility function to log response from sink, when sink reports error. This helps user find out why the sink is not working more quickly.

Vector configuration

I use a local ClickHouse server as sink, standard input as source:

[sources.in]
type = "stdin"
decoding.codec = "json"

[sinks.out]
type = "clickhouse"
inputs = ["in"]
endpoint = "http://localhost:8123" 
database = "default"
table = "test_table"
auth.strategy = "basic"
auth.user = "default"
auth.password = ""

How did you test this PR?

I spin up the test sink ClickHouse, which talks to Vector via HTTP.
I created test table with:

create table test_table(value Int64);

I configured Vector to write to that table. Then I spin up Vector, wrote data in stdin:

{"value":"some_value"}

Since the data cannot be converted to Int64, ClickHouse reported an error. Without this patch, Vector won't tell you why. With this patch, Vector clearly showed what caused the error:

$ ./target/debug/vector --config test.toml
2026-01-01T09:11:04.813273Z  INFO vector::app: Log level is enabled. level="info"
2026-01-01T09:11:04.814745Z  INFO vector::app: Loading configs. paths=["test.toml"]
2026-01-01T09:11:04.823813Z  INFO vector::sources::file_descriptors: Capturing stdin.
2026-01-01T09:11:05.221184Z  INFO vector::topology::running: Running healthchecks.
2026-01-01T09:11:05.221550Z  INFO vector: Vector has started. debug="true" version="0.53.0" arch="aarch64" revision=""
2026-01-01T09:11:05.221584Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2026-01-01T09:11:05.223805Z  INFO vector::topology::builder: Healthcheck passed.
{"value":"some_value"}
2026-01-01T09:11:12.763568Z  INFO vector_common::shutdown: All sources have finished.
2026-01-01T09:11:12.763684Z  INFO vector_common::shutdown: Internal log [All sources have finished.] is being suppressed to avoid flooding.
2026-01-01T09:11:12.763780Z  INFO vector::app: All sources have finished.
2026-01-01T09:11:12.763838Z  INFO vector: Vector has stopped.
2026-01-01T09:11:12.764996Z  INFO vector::topology::running: Shutting down... Waiting on running components. remaining_components="out" time_remaining="59 seconds left"
2026-01-01T09:11:12.881649Z ERROR sink{component_kind="sink" component_id=out component_type=clickhouse}:request{request_id=1}: vector::sinks::util::retries: Not retriable; dropping the request. reason="response status: 400 Bad Request, body: Code: 27. DB::Exception: Cannot parse input: expected '\"' before: 'some_value\"}\\n': (while reading the value of key value): (at row 1)\n: While executing ParallelParsingBlockInputFormat. (CANNOT_PARSE_INPUT_ASSERTION_FAILED) (version 25.11.2.24 (official build))\n"
2026-01-01T09:11:12.881979Z ERROR sink{component_kind="sink" component_id=out component_type=clickhouse}:request{request_id=1}: vector_common::internal_event::service: Service call failed. No retries or retries exhausted. error=None request_id=1 error_type="request_failed" stage="sending"
2026-01-01T09:11:14.493112Z ERROR sink{component_kind="sink" component_id=out component_type=clickhouse}:request{request_id=1}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=1 reason="Service call failed. No retries or retries exhausted."

Change Type

Bug fix
New feature
Non-functional (chore, refactoring, docs)
Performance

Is this a breaking change?

Yes
No

Does this PR include user facing changes?

Yes. Please add a changelog fragment based on our guidelines.
No. A maintainer will apply the no-changelog label to this PR.

References

Notes

Please read our Vector contributor resources.
Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
Some CI checks run only after we manually approve them.
- We recommend adding a pre-push hook, please see this template.
- Alternatively, we recommend running the following locally before pushing to the remote branch:
  - make fmt
  - make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
  - make test
After a review is requested, please avoid force pushes to help us review incrementally.
- Feel free to push as many commits as you want. They will be squashed into one before merging.
- For example, you can run git merge origin master and git push.
If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
run make build-licenses to regenerate the license inventory and commit the changes (if any). More details here.

github-actions · 2026-01-01T09:23:32Z

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

keuin · 2026-01-01T09:25:13Z

I have read the CLA Document and I hereby sign the CLA

thomasqueirozb

Hi @keuin, thanks for your PR! While this functionality is good I fear it may also add CPU overhead to users which see repeated http failures and could become quite spammy. I think that this should be gated behind a configuration option and disabled by default.

We can then include the configuration option over at https://vector.dev/guides/developer/debugging (file website/content/en/guides/developer/debugging.md) so that users are able to easily turn this on/off to debug issues.

thomasqueirozb · 2026-01-16T22:44:37Z

src/sinks/util/http.rs

+        "gzip" => Box::new(GzDecoder::new(body_bytes)),
+        "deflate" => Box::new(ZlibDecoder::new(body_bytes)),
+        "br" => Box::new(BrotliDecoder::new(body_bytes, BROTLI_INTERNAL_BUFFER_SIZE)),
+        "zstd" => Box::new(
+            ZstdDecoder::new(body_bytes).unwrap_or_else(|_| ZstdDecoder::new(body_bytes).unwrap()),
+        ),


I'm not sure these would work given we're reading only partial data

Yes, this works because the truncation is applied on decompressed byte stream, not the original compressed data (body_bytes). The decompressors can always read as many bytes as they need.

keuin · 2026-01-17T08:50:27Z

Hi @keuin, thanks for your PR! While this functionality is good I fear it may also add CPU overhead to users which see repeated http failures and could become quite spammy. I think that this should be gated behind a configuration option and disabled by default.

We can then include the configuration option over at https://vector.dev/guides/developer/debugging (file website/content/en/guides/developer/debugging.md) so that users are able to easily turn this on/off to debug issues.

I agree with you. This will indeed spam the log if there are repeated errors, because 1024 bytes is not short and will take multiple lines in screen. I will add a gate flag to disable this by default.

keuin · 2026-01-17T10:09:19Z

I gated this feature with sink-http-response, which will be enabled only if log level is set to DEBUG/TRACE or VECTOR_LOG=sink-http-response is set. Here is the updated test result, please have a look at it, thanks @thomasqueirozb :

Disabled (default behavior), no new logs appeared:

$ ./target/debug/vector --config config.toml 
2026-01-17T09:41:43.582429Z  INFO vector::app: Log level is enabled. level="info"
2026-01-17T09:41:43.586369Z  INFO vector::app: Loading configs. paths=["config.toml"]
2026-01-17T09:41:43.607713Z  INFO vector::sources::file_descriptors: Capturing stdin.
2026-01-17T09:41:44.033066Z  INFO vector::topology::running: Running healthchecks.
2026-01-17T09:41:44.035300Z  INFO vector: Vector has started. debug="true" version="0.53.0" arch="aarch64" revision=""
2026-01-17T09:41:44.035566Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2026-01-17T09:41:44.046181Z  INFO vector::topology::builder: Healthcheck passed.
{"value":"some_value"}

2026-01-17T09:42:22.921275Z ERROR sink{component_kind="sink" component_id=out component_type=clickhouse}:request{request_id=1}: vector::sinks::util::retries: Not retriable; dropping the request. reason="response status: 400 Bad Request"
2026-01-17T09:42:22.921548Z ERROR sink{component_kind="sink" component_id=out component_type=clickhouse}:request{request_id=1}: vector_common::internal_event::service: Service call failed. No retries or retries exhausted. error=None request_id=1 error_type="request_failed" stage="sending"
2026-01-17T09:42:22.921810Z ERROR sink{component_kind="sink" component_id=out component_type=clickhouse}:request{request_id=1}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=1 reason="Service call failed. No retries or retries exhausted."
2026-01-17T09:42:39.743511Z  INFO vector_common::shutdown: All sources have finished.
2026-01-17T09:42:39.743604Z  INFO vector_common::shutdown: Internal log [All sources have finished.] is being suppressed to avoid flooding.
2026-01-17T09:42:39.743797Z  INFO vector::app: All sources have finished.
2026-01-17T09:42:39.743878Z  INFO vector: Vector has stopped.
2026-01-17T09:42:39.746419Z  INFO vector::topology::running: Shutting down... Waiting on running components. remaining_components="out" time_remaining="59 seconds left"

Enabled (when sink-http-response is set to DEBUG/TRACE), the line containing sink (ClickHouse) HTTP response appeared.:

$ VECTOR_LOG=info,sink-http-response ./target/debug/vector --config config.toml 
2026-01-17T09:43:15.166057Z  INFO vector::app: Log level is enabled. level="info,sink-http-response"
2026-01-17T09:43:15.181685Z  INFO vector::app: Loading configs. paths=["config.toml"]
2026-01-17T09:43:15.244845Z  INFO vector::sources::file_descriptors: Capturing stdin.
2026-01-17T09:43:15.680642Z  INFO vector::topology::running: Running healthchecks.
2026-01-17T09:43:15.685249Z  INFO vector: Vector has started. debug="true" version="0.53.0" arch="aarch64" revision=""
2026-01-17T09:43:15.685564Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2026-01-17T09:43:15.703516Z  INFO vector::topology::builder: Healthcheck passed.
{"value":"some_value"}

2026-01-17T09:43:20.103295Z DEBUG sink{component_kind="sink" component_id=out component_type=clickhouse}:request{request_id=1}: sink-http-response: sink http response status=400 Bad Request body_preview=Code: 27. DB::Exception: Cannot parse input: expected '"' before: 'some_value"}\n': (while reading the value of key value): (at row 1)
: While executing ParallelParsingBlockInputFormat. (CANNOT_PARSE_INPUT_ASSERTION_FAILED) (version 25.12.3.21 (official build))

2026-01-17T09:43:20.103621Z ERROR sink{component_kind="sink" component_id=out component_type=clickhouse}:request{request_id=1}: vector::sinks::util::retries: Not retriable; dropping the request. reason="response status: 400 Bad Request"
2026-01-17T09:43:20.103885Z DEBUG sink{component_kind="sink" component_id=out component_type=clickhouse}:request{request_id=1}: sink-http-response: Internal log [sink http response] is being suppressed to avoid flooding.
2026-01-17T09:43:20.104347Z ERROR sink{component_kind="sink" component_id=out component_type=clickhouse}:request{request_id=1}: vector_common::internal_event::service: Service call failed. No retries or retries exhausted. error=None request_id=1 error_type="request_failed" stage="sending"
2026-01-17T09:43:20.104774Z ERROR sink{component_kind="sink" component_id=out component_type=clickhouse}:request{request_id=1}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=1 reason="Service call failed. No retries or retries exhausted."
2026-01-17T09:43:25.921723Z  INFO vector_common::shutdown: All sources have finished.
2026-01-17T09:43:25.921818Z  INFO vector_common::shutdown: Internal log [All sources have finished.] is being suppressed to avoid flooding.
2026-01-17T09:43:25.926691Z  INFO vector::app: All sources have finished.
2026-01-17T09:43:25.926800Z  INFO vector: Vector has stopped.
2026-01-17T09:43:25.929187Z  INFO vector::topology::running: Shutting down... Waiting on running components. remaining_components="in, out" time_remaining="59 seconds left"

keuin requested a review from a team as a code owner January 1, 2026 09:23

github-actions bot added the domain: sinks Anything related to the Vector's sinks label Jan 1, 2026

keuin changed the title ~~feat(sink): log response body when HTTP response failed~~ feat(sinks): log response body when HTTP response failed Jan 1, 2026

keuin force-pushed the feature/log-http-response-error branch 5 times, most recently from d27d01f to 35f0bf7 Compare January 1, 2026 15:25

thomasqueirozb reviewed Jan 16, 2026

View reviewed changes

thomasqueirozb added source: http_client Anything `http_client` source related meta: awaiting author Pull requests that are awaiting their author. sink: http Anything `http` sink related and removed source: http_client Anything `http_client` source related labels Jan 17, 2026

keuin force-pushed the feature/log-http-response-error branch from 35f0bf7 to 4b2e95f Compare January 17, 2026 09:44

github-actions bot removed the meta: awaiting author Pull requests that are awaiting their author. label Jan 17, 2026

keuin force-pushed the feature/log-http-response-error branch from 4b2e95f to 1468193 Compare January 17, 2026 10:04

keuin requested a review from a team as a code owner January 17, 2026 10:04

feat(sinks): log response body when HTTP response failed

2a0dc89

keuin force-pushed the feature/log-http-response-error branch from 1468193 to 2a0dc89 Compare January 17, 2026 10:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(sinks): log response body when HTTP response failed #24434

feat(sinks): log response body when HTTP response failed #24434

keuin commented Jan 1, 2026

Uh oh!

github-actions bot commented Jan 1, 2026 •

edited

Loading

Uh oh!

keuin commented Jan 1, 2026

Uh oh!

thomasqueirozb left a comment

Uh oh!

thomasqueirozb Jan 16, 2026

Uh oh!

keuin Jan 17, 2026

Uh oh!

keuin commented Jan 17, 2026

Uh oh!

keuin commented Jan 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

feat(sinks): log response body when HTTP response failed #24434

Are you sure you want to change the base?

feat(sinks): log response body when HTTP response failed #24434

Conversation

keuin commented Jan 1, 2026

Summary

Vector configuration

How did you test this PR?

Change Type

Is this a breaking change?

Does this PR include user facing changes?

References

Notes

Uh oh!

github-actions bot commented Jan 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

keuin commented Jan 1, 2026

Uh oh!

thomasqueirozb left a comment

Choose a reason for hiding this comment

Uh oh!

thomasqueirozb Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

keuin Jan 17, 2026

Choose a reason for hiding this comment

Uh oh!

keuin commented Jan 17, 2026

Uh oh!

keuin commented Jan 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions bot commented Jan 1, 2026 •

edited

Loading